Log-chromaticity clustering pipeline for use in an image process

ABSTRACT

In a first exemplary embodiment of the present invention, an automated, computerized method is provided for processing an image. According to a feature of the present invention, the method comprises the steps of providing an image file depicting an image defined by image locations, in a computer memory, generating a bi-illuminant chromaticity plane in a log color space for representing the image locations of the image in a log-chromaticity representation for the image, providing a set of estimates for the orientation of the bi-illuminant chromaticity plane and calculating an orientation for each one of the image locations as a function of the set of estimates for the orientation.

BACKGROUND OF THE INVENTION

Many significant and commercially important uses of modern computertechnology relate to images. These include image processing, imageanalysis and computer vision applications. In computer visionapplications, such as, for example, object recognition and opticalcharacter recognition, it has been found that a separation ofillumination and material aspects of an image can significantly improvethe accuracy of computer performance. Significant pioneer inventionsrelated to the illumination and material aspects of an image aredisclosed in U.S. Pat. No. 7,873,219 to Richard Mark Friedhoff, entitledDifferentiation Of Illumination And Reflection Boundaries and U.S. Pat.No. 7,672,530 to Richard Mark Friedhoff et al., entitled Method AndSystem For Identifying Illumination Flux In An Image (hereinafter theFriedhoff Patents).

SUMMARY OF THE INVENTION

The present invention provides an improvement and enhancement to thefundamental teachings of the Friedhoff Patents, and includes a methodand system comprising image techniques that accurately and correctlygenerate intrinsic images, including techniques to provide increasedaccuracy and precision in the determination of image characteristicsused in the generation of the intrinsic images.

In a first exemplary embodiment of the present invention, an automated,computerized method is provided for processing an image. According to afeature of the present invention, the method comprises the steps ofproviding an image file depicting an image defined by image locations,in a computer memory, generating a bi-illuminant chromaticity plane in alog color space for representing the image locations of the image in alog-chromaticity representation for the image, providing a set ofestimates for an orientation of the bi-illuminant chromaticity plane andgenerating a plurality of normal maps, each normal map providing anindication of at least one orientation, each at least one orientationcalculated as a function of the set of estimates for an orientation.

In a second exemplary embodiment of the present invention, a computersystem is provided. The computer system comprises a CPU and a memorystoring an image file containing an image defined by image locations.According to a feature of the present invention, the CPU is arranged andconfigured to execute a routine to generate a bi-illuminant chromaticityplane in a log color space for representing the image locations of theimage in a log-chromaticity representation for the image, provide a setof estimates for the orientation of the bi-illuminant chromaticity planeand generate a plurality of normal maps, each normal map providing anindication of at least one orientation, each at least one orientationcalculated as a function of the set of estimates for an orientation.

In a third exemplary embodiment of the present invention, a computerprogram product, disposed on a computer readable media is provided. Thecomputer program product includes computer executable process stepsoperable to control a computer to: provide an image file depicting animage defined by image locations, in a computer memory, generate abi-illuminant chromaticity plane in a log color space for representingthe image locations of the image in a log-chromaticity representationfor the image, provide a set of estimates for the orientation of thebi-illuminant chromaticity plane and generate a plurality of normalmaps, each normal map providing an indication of at least oneorientation, each at least one orientation calculated as a function ofthe set of estimates for an orientation.

In accordance with yet further embodiments of the present invention,computer systems are provided, which include one or more computersconfigured (e.g., programmed) to perform the methods described above. Inaccordance with other embodiments of the present invention,non-transitory computer readable media are provided which have storedthereon computer executable process steps operable to control acomputer(s) to implement the embodiments described above. The presentinvention contemplates a computer readable media as any product thatembodies information usable in a computer to execute the methods of thepresent invention, including instructions implemented as a hardwarecircuit, for example, as in an integrated circuit chip. The automated,computerized methods can be performed by a digital computer, analogcomputer, optical sensor, state machine, sequencer, integrated chip orany device or apparatus that can be designed or programmed to carry outthe steps of the methods of the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a computer system arranged and configuredto perform operations related to images.

FIG. 2 shows an n×m pixel array image file for an image stored in thecomputer system of FIG. 1.

FIG. 3 a is a flow chart for identifying Type C token regions in theimage file of FIG. 2, according to a feature of the present invention.

FIG. 3 b is an original image used as an example in the identificationof Type C tokens.

FIG. 3 c shows Type C token regions in the image of FIG. 3 b.

FIG. 3 d shows Type B tokens, generated from the Type C tokens of FIG. 3c, according to a feature of the present invention.

FIG. 4 is a flow chart for a routine to test Type C tokens identified bythe routine of the flow chart of FIG. 3 a, according to a feature of thepresent invention.

FIG. 5 is a graphic representation of a log color space chromaticityplane according to a feature of the present invention.

FIG. 6 is a flow chart for determining a list of colors depicted in aninput image.

FIG. 7 a is a flow chart for determining an orientation for alog-chromaticity space.

FIG. 7 b is a flow chart for an additional exemplary embodiment of thepresent invention, for determining an optimized orientation for alog-chromaticity space.

FIG. 7 c is a flow chart for a log-chromaticity normal optimizationtechnique, according to a feature of the present invention.

FIG. 7 d is a flow chart for implementing a log-chromaticity normaloptimization technique when a normal is estimated based upon a userselected lit-dark pairs of pixel blocks.

FIG. 7 e is a flow chart illustrating an entropy minimization techniqueaccording to a feature of the present invention.

FIG. 7 f is a flow chart showing the use of a system of linear equationsto estimate spatially varying normals.

FIG. 7 g shows an example of a token map having four tokens for use inthe system of linear equations of FIG. 7 f.

FIG. 7 h is a flow chart for a multi-clustering merge step.

FIG. 8 is a flow chart for determining log-chromaticity coordinates forthe colors of an input image, as determined through execution of theroutine of FIG. 6.

FIG. 9 is a flow chart for augmenting the log-chromaticity coordinates,as determined through execution of the routine of FIG. 8.

FIG. 10 is a flow chart for clustering the log-chromaticity coordinates,according to a feature of the present invention.

FIG. 10 a is an illustration of a grid for a spatial hash, according toa feature of the present invention.

FIG. 11 is a flow chart for assigning the log-chromaticity coordinatesto clusters determined through execution of the routine of FIG. 10.

FIG. 12 is a flow chart for detecting regions of uniform reflectancebased on the log-chromaticity clustering.

FIG. 13 is a representation of an [A] [x]=[b] matrix relationship usedto identify and separate illumination and material aspects of an image,according to a same-material constraint, for generation of intrinsicimages.

FIG. 14 illustrates intrinsic images including an illumination image anda material image corresponding to the original image of FIG. 3 b.

FIG. 15 is a flow chart for an edge preserving blur post processingtechnique applied to the intrinsic images illustrated in FIG. 14,according to a feature of the present invention.

FIG. 16 is a flow chart for an artifact reduction post processingtechnique applied to the intrinsic images illustrated in FIG. 14,according to a feature of the present invention.

FIG. 17 is a flow chart for a BIDR model enforcement post processingtechnique applied to the intrinsic images illustrated in FIG. 14,according to a feature of the present invention.

FIG. 18 is a graph in RGB color space showing colors for a material,from a fully shaded color value to a fully lit color value, as predictedby a bi-illuminant dichromatic reflection model.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Referring now to the drawings, and initially to FIG. 1, there is shown ablock diagram of a computer system 10 arranged and configured to performoperations related to images. A CPU 12 is coupled to a device such as,for example, a digital camera 14 via, for example, a USB port. Thedigital camera 14 operates to download images stored locally on thecamera 14, to the CPU 12. The CPU 12 stores the downloaded images in amemory 16 as image files 18. The image files 18 can be accessed by theCPU 12 for display on a monitor 20, or for print out on a printer 22.

Alternatively, the CPU 12 can be implemented as a microprocessorembedded in a device such as, for example, the digital camera 14 or arobot. The CPU 12 can also be equipped with a real time operating systemfor real time operations related to images, in connection with, forexample, a robotic operation or an interactive operation with a user.

As shown in FIG. 2, each image file 18 comprises an n×m pixel array.Each pixel, p, is a picture element corresponding to a discrete portionof the overall image. All of the pixels together define the imagerepresented by the image file 18. Each pixel comprises a digital valuecorresponding to a set of color bands, for example, red, green and bluecolor components (RGB) of the picture element. The present invention isapplicable to any multi-band image, where each band corresponds to apiece of the electro-magnetic spectrum. The pixel array includes n rowsof m columns each, starting with the pixel p (1,1) and ending with thepixel p(n, m). When displaying or printing an image, the CPU 12retrieves the corresponding image file 18 from the memory 16, andoperates the monitor 20 or printer 22, as the case may be, as a functionof the digital values of the pixels in the image file 18, as isgenerally known.

In an image operation, the CPU 12 operates to analyze the RGB values ofthe pixels of a stored image file 18 to achieve various objectives, suchas, for example, to identify regions of an image that correspond to asingle material depicted in a scene recorded in the image file 18. Afundamental observation underlying a basic discovery of the presentinvention, is that an image comprises two components, material andillumination. All changes in an image are caused by one or the other ofthese components. A method for detecting of one of these components, forexample, material, provides a mechanism for distinguishing material orobject geometry, such as object edges, from illumination and shadowboundaries.

Such a mechanism enables techniques that can be used to generateintrinsic images. The intrinsic images correspond to an original image,for example, an image depicted in an input image file 18. The intrinsicimages include, for example, an illumination image, to capture theintensity and color of light incident upon each point on the surfacesdepicted in the image, and a material reflectance image, to capturereflectance properties of surfaces depicted in the image (the percentageof each wavelength of light a surface reflects). The separation ofillumination from material in the intrinsic images provides the CPU 12with images optimized for more effective and accurate furtherprocessing.

Pursuant to a feature of the present invention, processing is performedat a token level. A token is a connected region of an image wherein thepixels of the region are related to one another in a manner relevant toidentification of image features and characteristics such as anidentification of materials and illumination. The pixels of a token canbe related in terms of either homogeneous factors, such as, for example,close correlation of color among the pixels, or inhomogeneous factors,such as, for example, differing color values related geometrically in acolor space such as RGB space, commonly referred to as a texture. Thepresent invention utilizes spatio-spectral information relevant tocontiguous pixels of an image depicted in an image file 18 to identifytoken regions. The spatio-spectral information includes spectralrelationships among contiguous pixels, in terms of color bands, forexample the RGB values of the pixels, and the spatial extent of thepixel spectral characteristics relevant to a single material.

According to one exemplary embodiment of the present invention, tokensare each classified as either a Type A token, a Type B token or a Type Ctoken. A Type A token is a connected image region comprising contiguouspixels that represent the largest possible region of the imageencompassing a single material in the scene (uniform reflectance). AType B token is a connected image region comprising contiguous pixelsthat represent a region of the image encompassing a single material inthe scene, though not necessarily the maximal region of uniformreflectance corresponding to that material. A Type B token can also bedefined as a collection of one or more image regions or pixels, all ofwhich have the same reflectance (material color) though not necessarilyall pixels which correspond to that material color. A Type C tokencomprises a connected image region of similar image properties among thecontiguous pixels of the token, where similarity is defined with respectto a noise model for the imaging system used to record the image.

Referring now to FIG. 3 a, there is shown a flow chart for identifyingType C token regions in the scene depicted in the image file 18 of FIG.2, according to a feature of the present invention. Type C tokens can bereadily identified in an image, utilizing the steps of FIG. 3 a, andthen analyzed and processed to construct Type B tokens, according to afeature of the present invention.

A 1^(st) order uniform, homogeneous Type C token comprises a singlerobust color measurement among contiguous pixels of the image. At thestart of the identification routine, the CPU 12 sets up a region map inmemory. In step 100, the CPU 12 clears the region map and assigns aregion ID, which is initially set at 1. An iteration for the routine,corresponding to a pixel number, is set at i=0, and a number for an N×Npixel array, for use as a seed to determine the token, is set an initialvalue, N=N_(start). N_(start) can be any integer >0, for example it canbe set at set at 11 or 15 pixels.

At step 102, a seed test is begun. The CPU 12 selects a first pixel,i=1, pixel (1, 1) for example (see FIG. 2), the pixel at the upper leftcorner of a first N×N sample of the image file 18. The pixel is thentested in decision block 104 to determine if the selected pixel is partof a good seed. The test can comprise a comparison of the color value ofthe selected pixel to the color values of a preselected number of itsneighboring pixels as the seed, for example, the N×N array. The colorvalues comparison can be with respect to multiple color band values (RGBin our example) of the pixel. If the comparison does not result inapproximately equal values (within the noise levels of the recordingdevice) for the pixels in the seed, the CPU 12 increments the value of i(step 106), for example, i=2, pixel (1, 2), for a next N×N seed sample,and then tests to determine if i=i_(max) (decision block 108).

If the pixel value is at i_(max), a value selected as a threshold fordeciding to reduce the seed size for improved results, the seed size, N,is reduced (step 110), for example, from N=15 to N=12. In an exemplaryembodiment of the present invention, i_(max) can be set at a number ofpixels in an image ending at pixel (n, m), as shown in FIG. 2. In thismanner, the routine of FIG. 3 a parses the entire image at a first valueof N before repeating the routine for a reduced value of N.

After reduction of the seed size, the routine returns to step 102, andcontinues to test for token seeds. An N_(stop) value (for example, N=2)is also checked in step 110 to determine if the analysis is complete. Ifthe value of N is at N_(stop), the CPU 12 has completed a survey of theimage pixel arrays and exits the routine.

If the value of i is less than i_(max), and N is greater than N_(stop),the routine returns to step 102, and continues to test for token seeds.

When a good seed (an N×N array with approximately equal pixel values) isfound (block 104), the token is grown from the seed. In step 112, theCPU 12 pushes the pixels from the seed onto a queue. All of the pixelsin the queue are marked with the current region ID in the region map.The CPU 12 then inquires as to whether the queue is empty (decisionblock 114). If the queue is not empty, the routine proceeds to step 116.

In step 116, the CPU 12 pops the front pixel off the queue and proceedsto step 118. In step 118, the CPU 12 marks “good’ neighbors around thesubject pixel, that is neighbors approximately equal in color value tothe subject pixel, with the current region ID. All of the marked goodneighbors are placed in the region map and also pushed onto the queue.The CPU 12 then returns to the decision block 114. The routine of steps114, 116, 118 is repeated until the queue is empty. At that time, all ofthe pixels forming a token in the current region will have beenidentified and marked in the region map as a Type C token.

When the queue is empty, the CPU 12 proceeds to step 120. At step 120,the CPU 12 increments the region ID for use with identification of anext token. The CPU 12 then returns to step 106 to repeat the routine inrespect of the new current token region.

Upon arrival at N=N_(stop), step 110 of the flow chart of FIG. 3 a, orcompletion of a region map that coincides with the image, the routinewill have completed the token building task. FIG. 3 b is an originalimage used as an example in the identification of tokens. The imageshows areas of the color blue and the blue in shadow, and of the colorteal and the teal in shadow. FIG. 3 c shows token regions correspondingto the region map, for example, as identified through execution of theroutine of FIG. 3 a (Type C tokens), in respect to the image of FIG. 3b. The token regions are color coded to illustrate the token makeup ofthe image of FIG. 3 b, including penumbra regions between the full colorblue and teal areas of the image and the shadow of the colored areas.

While each Type C token comprises a region of the image having a singlerobust color measurement among contiguous pixels of the image, the tokenmay grow across material boundaries. Typically, different materialsconnect together in one Type C token via a neck region often located onshadow boundaries or in areas with varying illumination crossingdifferent materials with similar hue but different intensities. A neckpixel can be identified by examining characteristics of adjacent pixels.When a pixel has two contiguous pixels on opposite sides that are notwithin the corresponding token, and two contiguous pixels on oppositesides that are within the corresponding token, the pixel is defined as aneck pixel.

FIG. 4 shows a flow chart for a neck test for Type C tokens. In step122, the CPU 12 examines each pixel of an identified token to determinewhether any of the pixels under examination forms a neck. The routine ofFIG. 4 can be executed as a subroutine directly after a particular tokenis identified during execution of the routine of FIG. 3 a. All pixelsidentified as a neck are marked as “ungrowable.” In decision block 124,the CPU 12 determines if any of the pixels were marked.

If no, the CPU 12 exits the routine of FIG. 4 and returns to the routineof FIG. 3 a (step 126).

If yes, the CPU 12 proceeds to step 128 and operates to regrow the tokenfrom a seed location selected from among the unmarked pixels of thecurrent token, as per the routine of FIG. 3 a, without changing thecounts for seed size and region ID. During the regrowth process, the CPU12 does not include any pixel previously marked as ungrowable. After thetoken is regrown, the previously marked pixels are unmarked so thatother tokens may grow into them.

Subsequent to the regrowth of the token without the previously markedpixels, the CPU 12 returns to step 122 to test the newly regrown token.Neck testing identifies Type C tokens that cross material boundaries,and regrows the identified tokens to provide single material Type Ctokens suitable for use in creating Type B tokens.

FIG. 3 d shows Type B tokens generated from the Type C tokens of FIG. 3c, according to a feature of the present invention. The presentinvention provides a novel exemplary technique using log-chromaticityclustering, for constructing Type B tokens for an image file 18. Logchromaticity is a technique for developing an illumination invariantchromaticity space.

A method and system for separating illumination and reflectance using alog-chromaticity representation is disclosed in U.S. Pat. No. 7,596,266,which is hereby expressly incorporated by reference. The techniquestaught in U.S. Pat. No. 7,596,266 can be used to provide illuminationinvariant log-chromaticity representation values for each color of animage, for example, as represented by Type C tokens. Logarithmic valuesof the color band values of the image pixels are plotted on a log-colorspace graph. The logarithmic values are then projected to alog-chromaticity projection plane oriented as a function of abi-illuminant dichromatic reflection model (BIDR model), to provide alog-chromaticity value for each pixel, as taught in U.S. Pat. No.7,596,266. The BIDR Model predicts that differing color measurementvalues fall within a cylinder in RGB space, from a dark end (in shadow)to a bright end (lit end), along a positive slope, when the color changeis due to an illumination change forming a shadow over a single materialof a scene depicted in the image.

FIG. 5 is a graphic representation of a log color space, bi-illuminantchromaticity plane according to a feature of the invention disclosed inU.S. Pat. No. 7,596,266. The alignment of the chromaticity plane isdetermined by a vector N, normal to the chromaticity plane, and definedas N=log(Bright_(vector))−log(Dark_(vector))=log(1+1/S_(vector)). Theco-ordinates of the plane, u, v can be defined by a projection of thegreen axis onto the chromaticity plane as the u axis, and the crossproduct of u and N being defined as the v axis. In our example, each logvalue for the materials A, B, C is projected onto the chromaticityplane, and will therefore have a corresponding u, v co-ordinate value inthe plane that is a chromaticity value, as shown in FIG. 5.

Thus, according to the technique disclosed in U.S. Pat. No. 7,596,266,the RGB values of each pixel in an image file 18 can be mapped by theCPU 12 from the image file value p(n, m, R, G, B) to a log value, then,through a projection to the chromaticity plane, to the corresponding u,v value, as shown in FIG. 5. Each pixel p(n, m, R, G, B) in the imagefile 18 is then replaced by the CPU 12 by a two dimensional chromaticityvalue: p(n, m, u, v), to provide a chromaticity representation of theoriginal RGB image. In general, for an N band image, the N color valuesare replaced by N−1 chromaticity values. The chromaticity representationis a truly accurate illumination invariant representation because theBIDR model upon which the representation is based, accurately andcorrectly represents the illumination flux that caused the originalimage.

According to the invention disclosed and claimed in related applicationSer. No. 12/927,244, filed Nov. 10, 2010, entitled System and Method forIdentifying Complex Tokens in an Image (expressly incorporated byreference herein and hereinafter referred to as “related invention”),published as US 2012/0114232, log-chromaticity values are calculated foreach color depicted in an image file 18 input to the CPU 12 foridentification of regions of the uniform reflectance (Type B tokens).For example, each pixel of a Type C token will be of approximately thesame color value, for example, in terms of RGB values, as all the otherconstituent pixels of the same Type C token, within the noise level ofthe equipment used to record the image. Thus, an average of the colorvalues for the constituent pixels of each particular Type C token can beused to represent the color value for the respective Type C token in thelog-chromaticity analysis.

FIG. 6 is a flow chart for determining a list of colors depicted in aninput image, for example, an image file 18. In step 200, an input imagefile 18 is input to the CPU 12 for processing. In steps 202 and 204, theCPU 12 determines the colors depicted in the input image file 18. Instep 202, the CPU 12 calculates an average color for each Type C tokendetermined by the CPU 12 through execution of the routine of FIG. 3 a,as described above, for a list of colors. The CPU 12 can be operated tooptionally require a minimum token size, in terms of the number ofconstituent pixels of the token, or a minimum seed size (the N×N array)used to determine Type C tokens according to the routine of FIG. 3 a,for the analysis. The minimum size requirements are implemented toassure that color measurements in the list of colors for the image arean accurate depiction of color in a scene depicted in the input image,and not an artifact of blend pixels.

Blend pixels are pixels between two differently colored regions of animage. If the colors between the two regions are plotted in RGB space,there is a linear transition between the colors, with each blend pixel,moving from one region to the next, being a weighted average of thecolors of the two regions. Thus, each blend pixel does not represent atrue color of the image. If blend pixels are present, relatively smallType C tokens, consisting of blend pixels, can be identified for areasof an image between two differently colored regions. By requiring a sizeminimum, the CPU 12 can eliminate tokens consisting of blend pixel fromthe analysis.

In step 204, the CPU 12 can alternatively collect colors at the pixellevel, that is, the RGB values of the pixels of the input image file 18,as shown in FIG. 2. The CPU 12 can be operated to optionally requireeach pixel of the image file 18 used in the analysis to have a minimumstability or local standard deviation via a filter output, for a moreaccurate list of colors. For example, second derivative energy can beused to indicate the stability of pixels of an image.

In this approach, the CPU 12 calculates a second derivative at eachpixel, or a subset of pixels disbursed across the image to cover allillumination conditions of the image depicted in an input image file 18,using a Difference of Gaussians, Laplacian of Gaussian, or similarfilter. The second derivative energy for each pixel examined can then becalculated by the CPU 12 as the average of the absolute value of thesecond derivative in each color band (or the absolute value of thesingle value in a grayscale image), the sum of squares of the values ofthe second derivatives in each color band (or the square of the singlevalue in a grayscale image), the maximum squared second derivative valueacross the color bands (or the square of the single value in a grayscaleimage), or any similar method. Upon the calculation of the secondderivative energy for each of the pixels, the CPU 12 analyzes the energyvalues of the pixels. There is an inverse relationship between secondderivative energy and pixel stability, the higher the energy, the lessstable the corresponding pixel.

In step 206, the CPU 12 outputs a list or lists of color (afterexecuting one or both of steps 202 and/or 204). According to a featureof the related invention, all of the further processing can be executedusing the list from either step 202 or 204, or vary the list used (oneor the other of the lists from steps 202 or 204) at each subsequentstep.

FIG. 7 a is a flow chart for determining an orientation for alog-chromaticity representation, according to a feature of the relatedinvention. For example, the CPU 12 determines an orientation for thenormal N, for a log-chromaticity plane, as shown in FIG. 5. In step 210,the CPU 12 receives a list of colors for an input file 18, such as alist output in step 206 of the routine of FIG. 6. In step 212, the CPU12 determines an orientation for a log-chromaticity plane.

As taught in U.S. Pat. No. 7,596,266, and as noted above, orientation ofthe chromaticity plane is represented by N, N being a vector normal tothe chromaticity representation, for example, the chromaticity plane ofFIG. 5. The orientation is estimated by the CPU 12 thorough execution ofany one of several techniques. For example, the CPU 12 can determineestimates based upon entropy minimization, manual selection oflit/shadowed regions of a same material by a user or the use of acharacteristic spectral ratio (which corresponds to the orientation N)for an image of an input image file 18, as fully disclosed in U.S. Pat.No. 7,596,266.

For a higher dimensional set of colors, for example, an RYGB space (red,yellow, green, blue), the log-chromaticity normal, N, defines asub-space with one less dimension than the input space. Thus, in thefour dimensional RYGB space, the normal N defines a three dimensionallog-chromaticity space. When the four dimensional RYGB values areprojected into the three dimensional log-chromaticity space, theprojected values within the log-chromaticity space are unaffected byillumination variation.

In step 214, the CPU 12 outputs an orientation for the normal N. Asillustrated in the example of FIG. 5, the normal N defines anorientation for a u, v plane in a three dimensional RGB space.

According to further exemplary embodiments of the present invention,processing techniques are implemented to provide increased accuracy andprecision in the determination of the normal, N, and, further, tocalculate spatially varying log-chromaticity normals, to account forvarying conditions that can exist in a scene depicted in an image file18. FIG. 7 b shows a flow chart for improving accuracy in the generationof illumination invariant (same reflectance) log-chromaticityrepresentation values for each color of the image using one or morenormals, N, to orient the log-chromaticity plane.

In step 1000, an image file 18 is input to the CPU 12, for example, theinput file 18 processed in the routine of FIG. 6. In step 1002, one ormore values (1−K) for the normal N are either automatically estimated bythe CPU 12 and/or provided by user input, as shown in sub-steps 1002a-d. Each normal is associated with corresponding pixel positions forthe pixels used to calculate the respective normal, N, for example, fromamong pixels p(1, 1) to p(n, m) of an image file 18 being processed, asshown in FIG. 2.

To implement sub-step 1002 a, a user designates, for example, via atouch screen action, one or more sets of lit-dark pairs of pixel blocks,the pairs each corresponding to lit and shadowed regions of a samematerial, respectively, depicted in the image of the image file 18 beingprocessed. Each pixel block includes, for example, an n×n array ofpixels. In sub-steps 1002 b-d, the CPU 12 operates to automaticallyestimate one or more values for the normal, N.

For example, in step 1002 b, the CPU 12 operates to identify lineartokens that are then used to estimate normal values. A linear token is anonhomogeneous token comprising a connected region of the image whereinadjacent pixels of the region have differing color measurement valuesthat fall within a cylinder in RGB space, from a dark end (in shadow) toa bright end (lit end) of a single material, along a positive slope(see, for example, FIG. 18, showing colors for a material, from a fullyshaded color value to a fully lit color value, as predicted by abi-illuminant dichromatic reflection model). The cylinder configurationis predicted by the bi-illuminant dichromatic reflection model (BIDRmodel).

As described above, the BIDR model predicts the correct colors for amaterial, in a shadow penumbra, from full shadow to fully lit. Eachlinear token, therefore, provides a candidate image region that likelycorresponds to a set of pixels extending through a penumbra across asingle material depicted in the image.

U.S. Pat. No. 7,995,058 discloses a technique for analyzing pixels of animage to identify contiguous pixels forming linear tokens throughout theimage. In step 1002 b, the CPU 12 is operated to execute the techniquetaught in U.S. Pat. No. 7,995,058, to identify a set of linear tokens inthe image being processed.

As noted above, the orientation for the log-chromaticity plane isdefined as N=log(1+1/S_(vector)), wherein S_(vector) is a characteristicspectral ratio defined asS_(vector)=Dark_(vector)/Bright_(vector)−Dark_(vector)), with thevectors corresponding to, for example, the RGB values for a fully litpixel (Bright) and a pixel in full shadow (Dark), respectively, for amaterial (see U.S. Pat. No. 7,596,266). Accordingly, the characteristicspectral ratio(s) for calculating a normal(s), N, for thelog-chromaticity plane can be based upon color information provided byeither one or both of the lit-dark pairs of pixel blocks selected by auser (step 1002 a) and/or the pixels defining the linear tokensidentified by the CPU 12 (step 1002 b).

In the case of the lit-dark pairs of pixel blocks selected by a user,the CPU 12 can calculate an average color value, for example, a median,for each n×n block of pixels for each lit and dark pair selected by auser. The result is an average Bright color and Dark color for eachselected pair. The CPU 12 then proceeds to use the average values tocalculate an S_(vector) based N value for each pair executing theN=log(1+1/S_(vector)) equation.

In the case of the identified linear tokens, in each case, the CPU 12fits a line to a plot, for example, in an RGB space, of the pixels ofthe respective linear token (see, for example, FIG. 18). The slope ofthe line can be used to represent an S_(vector). As in the previousexample, the CPU 12 then proceeds to use the line slope information tocalculate an S_(vector) based N value for each linear token.

Steps 1002 c and d provide additional automatic methods for calculatinga log-chromaticity plane orientation. In step 1002 c, the automaticcalculation is based upon X-junctions, and in step 1002 d, the automaticcalculation is based upon Type B tokens.

An X-junction is a region of an image wherein an illumination boundarycrosses a material object boundary. Accordingly, each X-junctionincludes two materials, with each material including lit and shadowedregions, for example, as depicted by four adjacent Type C tokens. Type Ctokens are identified by the CPU 12 via execution of the routine of FIG.3 a, as described above (step 1004).

U.S. Pat. No. 7,672,530 teaches a technique for automaticallyidentifying X-junctions in an image by analysis of token neighborrelationships indicative of spatio-spectral features of an image. TheCPU 12 is operated to implement the technique of U.S. Pat. No. 7,672,530to perform iterations through Type C token neighbor relationships toidentify a region where spectral ratios indicate a crossing ofillumination and material object boundaries (for example spectral ratiosbased upon Dark_(vector)/(Bright_(vector)−Dark_(vector)), wherein thevectors are sample pixels from each side of a boundary betweenneighboring Type C tokens).

An average, for example, a median, for the spectral ratios along theillumination boundary of each identified X-junction is used as theS_(vector) to calculate a normal N for each X-junction.

In step 1002 d the automatic calculation is based upon Type B tokens,for example, as identified by the CPU 12 through execution of theroutines of FIGS. 8-12, as will be described below. As noted above, eachType B token is a connected image region comprising contiguous pixelsthat represent a region of the image encompassing a single material inthe scene. Thus, each type B token potentially covers a region of theimage wherein one material extends from a fully lit end to an end infull shadow. Accordingly, a Type B token can be used to determine anS_(vector) for use in the calculation of a normal N.

For example, a Bright pixel for use in the calculation of an S_(vector)can be selected from the 95^(th) percentile among the pixels of a Type Btoken, and a Dark pixel selected from the 5^(th) percentile.

Each normal, 1−K, based upon either a user selection and/or calculatedby the CPU 12, is associated with pixel positions P corresponding to thepixels used to calculate the respective normal, N. For each normal basedupon a user-selected lit-dark pair, P includes the pixels of each n×narray of pixels of each selected lit and dark pixel block. For each ofthe normals based upon linear tokens, P includes the pixels of therespective linear token. For each of the normals based upon anX-junction, P includes the pixels of the Type C tokens defining arespective X-junction. For each of the normals based upon Type B tokens,P includes the pixels of the respective Type B token.

In step 1006, the CPU 12 operates to optimize each of the 1−K normals.Each of the 1−K normals is estimated utilizing relatively few of thepixels (for example, the pixel positions P) of the entire image. FIG. 7c is a flow chart for implementing a log-chromaticity normaloptimization technique (step 1006), according to a feature of thepresent invention, to provide a more robust estimate of each normalrelative to the entire set of pixels forming the image being processed.The CPU 12 executes the routine of FIG. 7 c once for each of the 1−Knormals selected by a user and/or calculated by the CPU 12, to optimizeeach one of the respective 1−K normals relative to all of the pixels ofthe image being processed.

In step 2000, one of the 1−K normals, is input to the CPU 12, (forexample, as estimated by one of the example techniques (corresponding tosteps 1002 a-c) (shown next to step 2000, in FIG. 7 c)), together withthe pixel positions P used to calculate the respective normal.

In step 2002 the CPU 12 calculates a weight w for each pixel of theimage being processed I, (an image file 18, also input to the CPU 12(step 2002 a)). The weight for each pixel of the image, in the exemplaryembodiment of the present invention, is determined as a function of eachof a spatial distance between the respective pixel and the set of pixelsP used to calculate the respective normal and a spectral distancebetween the pixel and the set of pixels P.

In a general case for the exemplary embodiment of the present invention,the weight w at a particular pixel p_(i), in the image I, relative to P,is computed based upon a set of M distance functions (M=1−k),d_(k)=(p_(i), P), between the pixel p_(i) and P.

When the k value for M equals two, for example, the spatial distance andspectral distance functions of the exemplary embodiment, a d_(k)(p_(i),P) value for each pixel corresponds to one of a d_(spatial) distancevalue and a d_(spectral) distance value for that pixel. Eachd_(k)(p_(i), P) value is calculated for each pixel p_(i) of the image bythe CPU 12. The spatial distance value measures a Euclidean norm in thespatial extent of the image, as shown, for example, in FIG. 2. Thespectral distance measures a distance in a color space, such as, forexample, the log color space shown in FIG. 5.

For the spatial distance: d_(spatial)(p_(i), P)=min (pεP)∥p_(i)−p∥₂wherein min (pεP) is the closest pixel in P to the pixel p_(i) in termsof a Euclidean distance, and ∥p_(i)−p∥₂ denotes the Euclidean norm.

For the spectral distance: d_(spectral)(p_(i),P)=min(pεP)∥I(p_(i))−I(p)∥₂ wherein min (pεP) is the closest pixel in Pto the pixel p_(i) in terms of spectral distance in a log-RGB space,I(p_(i)) is the log-RGB value and ∥I(p_(i))−I(p)∥₂ is the norm of thespectral distance.

Upon the calculation of a spatial distance and spectral distance foreach pixel, the CPU 12 operates to determine a weight for each pixel asa function of the calculated distances. Each of the spatial and spectraldistances for each pixel is converted by the CPU 12 to a weightw_(k,i)ε|0, 1|, for example, by implementing a soft threshold function,such as a sigmoid:w _(k,i)(d _(k)(p _(i) ,P))=1/1+exp(−β_(k)(d _(k)(p _(i) ,P)−τ_(k)))wherein β_(k) and τ_(k) are parameters defining the softness andposition of the threshold function, respectively. Thus, in the exemplaryembodiment of the present invention, when M=2, for each pixel p, aweight is calculated based upon the spatial distance for the pixel:w_(spatial)(d_(spatial)(p_(i), P)) and an additional weight iscalculated based upon the spectral distance:w_(spectral)(d_(spectral)(p_(i), P)).

In the exemplary embodiment of the present invention, the parametervalues can be set, as follows:

β_(spatial)=0.5

τ_(spatial)=10

β_(spectral)=5

τ_(spectral)=1

A single weight w_(i), for each particular pixel p_(i) is calculated asa function of the corresponding spatial and spectral weights, asfollows:w _(i)=Π(k=1,M)w _(k,i)(d _(k)(p _(i) ,P))

When M=2, w_(i)=w_(spatial)(d_(spatial)(p_(i),P))×w_(spectral)(d_(spectral)(p_(i), P))

In the case of a normal calculated based upon user selected lit-darkpairs of pixel blocks, a different algorithm can be implemented todetermine the spatial weight of each pixel. The concept of the weightalgorithm used when the estimate for the normal is based upon userselected lit, dark pairs relates to the fact that a line extendingbetween the centroid of the selected lit pixel block and the centroid ofthe selected dark pixel block provides more candidate pixels P_(l) (theadditional pixels forming the line) for the pixel positions P, resultingin a more robust estimate for the weight of the normal. FIG. 7 d is aflow chart for implementing a log-chromaticity normal optimizationtechnique when the normal is estimated based upon user selected lit-darkpairs of pixel blocks.

In steps 2010 a, b, a set of lit, dark n×n pixel blocks, as selected bya user, and the image being processed, I are input to the CPU 12.

In step 2012, the CPU 12 operates to fit a sigmoid on a line l extendingbetween the centroid (p_(lit-centroid)) of the lit n×n pixel block(P_(lit)) and the centroid (p_(dark-centroid)) of the dark n×n pixelblock (P_(dark)). Ideally, a user selects lit/dark pairs that are litand shadowed regions of a same material. However, a user can selectpairs of pixel blocks that are at different materials within the image.An analysis based upon a sigmoid fit can be used to verify that the lineis crossing a shadow boundary extending over a single material.

In an exemplary embodiment of the present invention, a parametric modelof illumination is used to determine if the line l extending between thecentroid of the lit pixel block and the centroid of the dark pixel blockoverlaps a shadow boundary extending across a single material. Forexample, the model for an illumination transition is a sigmoid σ:σ(x)=α/1+exp(−γ(x−x_(cen)))+s

wherein x is a point along the line l, α is a scale factor, γ is theslope of the sigmoid (rate of change in intensity with respect to the xposition), x_(cen) is the offset along l and s is an intensity offset.

Fitting the model to find the parameters (e.g. α and γ) that minimizethe difference between the model and the observed image data, can beimplemented via a non-linear optimization procedure such as, forexample, the Levenberg-Marquardt algorithm. A fit quality q for the setof pixels P_(l) forming the line l can be defined to be inverselyproportional to the residual between the final fitted model and theoriginal image data I. For example, the CPU 12 can use the root meansquared error measure:

$q = {- \sqrt{\frac{1}{P_{l}}{\sum\limits_{k \in P_{l}}{\sum\limits_{b \in {\{{R,G,B}\}}}\left( {{\sigma_{b}(k)} - {I_{b}(k)}} \right)^{2}}}}}$

In the decision block 2014, the CPU 12 determines if the fit quality qcalculated in step 2012, is greater than a threshold value, indicatingthat the line crosses a shadow boundary over a single material. If yes,the CPU 12 proceeds to step 2016 to compute a distance, d_(line), tomeasure a spatial distance from each pixel p_(i) in the image beingprocessed I to the line l, as follows:

first, calculate the quantity δ:δ=(p _(i) −p _(lit-centroid))×(p _(dark-centroid) −p _(lit-centroid))/∥p_(dark-centroid) −p _(lit-centroid)∥when δ<0, then d_(line)(p_(i), P_(l))=∥p_(i)−p_(lit-centroid)∥when 0≦δ≦1, then:d _(line)(p _(i) ,P _(l))=|(p _(lit-centroid) −P _(dark-centroid))×(p_(dark-centroid) −p _(i))|/|(p _(lit-centroid) −p _(dark-centroid))|when δ≦1, then: d_(line)(p_(i), P_(l))=∥p_(i)−p_(dark-centroid)∥

If no, the CPU 12 sets d_(line)(p_(i), P_(l))=∞, for each pixel of theimage I (step 2018).

After the performance of either steps 2016 or 2018, the CPU 12 proceedsto step 2020. In step 2020, the CPU 12 calculates a distanced_(point)(p_(i), P), to measure the spatial distance between each pixelp_(i) and P (the pixels of P_(lit) and P_(dark) without the pixels ofthe line l), as follows: d_(point)(p_(i), P)=min(pεp_(lit),P_(dark))∥p_(i)−p∥₂ (corresponding to the spatial distancecalculated in the general case, as described above).

In step 2022, the CPU 12 computes the spatial distance,d_(spatial)(p_(i), P), for each pixel, as a function of d_(line)(p_(i),P_(l)) (calculated in steps 2016 or 2018) and d_(point)(p_(i), P)calculated in step 2020, as follows:d _(spatial)(p _(i) ,P)=min(d _(line)(p _(i) ,P _(l)),d _(point)(p _(i),P)).wherein min (d_(line)(p_(i), P_(l)), d_(point)(p_(i), P)) is the minimumof the two distances.

In steps 2024 and 2026, the CPU 12 computes and outputs a weight foreach pixel, according to the steps executed by the CPU 12 in the generalcase, described above, using the spatial distance, as calculated for thecase of a user-selected normal, and the spectral distance, as calculatedin the general case.

Referring once again to FIG. 7 c, after the performance of step 2002, tocompute a weight for each pixel, the CPU 12 proceeds to step 2004. Instep 2004, the CPU 12 finds an optimized normal based upon a method tominimize a weighted entropy, using the weight calculation from step2002. In step 2004 a, a list of colors based upon Type C tokens, forexample, as determined in step 202 of FIG. 6, is input to the CPU 12,for use in the weighted entropy minimization technique.

FIG. 7 e is a flow chart illustrating an entropy minimization technique,according to a feature of the present invention. In step 2500, the listof colors (step 2004 a of FIG. 7 c) and an initial normal value (one ofthe 1−K estimates) are input to the CPU 12. In step 2502, the CPU 12operates to project each color from the list of colors to a chromaticityplane defined by the normal N. The projection step is executed as perthe routine of FIG. 8, as described in detail below.

In step 2504, the CPU 12 operates to estimate the entropy of theprojected colors. Entropy is inversely proportional to order, the lowerthe entropy, the higher the order of the system under review. At anoptimal orientation for the chromaticity plane, all bright and darkpixel pairs for a single material depicted in an image file 18, shouldproject to the same point on the chromaticity plane, a high order, orlow entropy state. The entropy of the projection of points onto theplane is a measure of how well the pairs line up for all materials in animage file 18, for example, the image file 18 for the image beingprocessed I. The lower the entropy, the higher the order of thechromaticity plane, and, thus, the more accurate the projections ofbright/dark pairs.

U.S. Pat. No. 7,596,266, teaches a method for finding an optimalalignment for the normal N for the chromaticity plane, by utilizing anentropy minimization technique. As described in detail in U.S. Pat. No.7,596,266, at each orientation selected for the chromaticity plane, ahistogram for the chromaticity plane shows the distribution of log colorspace projections among a grid of bins. The wider the distributionacross the plane of the histogram, the higher the entropy.

According to a feature of the present invention, the entropy equationbased upon the bins of a histogram (H) is computed as a function of theweight calculations performed in step 2002 of FIG. 7 c, as follows:H=Σ _(b) w _(b) p _(b) log p _(b)/Σ_(b) w _(b),wherein p_(b) is the value of each bin of the histogram, calculated toprovide an indication of the percentage of the distribution of log RGBvalues across the chromaticity plane in the bin, and w_(b) is the sum ofthe weights of the pixels located in a bin (one w_(b) sum value for eachbin).

In step 2506, the CPU 12 executes a search strategy for candidate valuesfor the normal, after the entropy calculation based upon the initialnormal estimate (one of the 1−K normal estimates). The CPU 12 canexecute any known search technique to select a series of orientationsfor the chromaticity plane, relative to the initial value, andthereafter, select the orientation for the plane having the lowestentropy. Such known search techniques include, for example, exhaustivesearch, univariate search, and simulated annealing search methodsdescribed in the literature. For example, the univariate searchtechnique is described in Hooke & Jeeves, “Direct Search Solution ofNumerical and Statistical Problems,” Journal of the ACM, Vol. 8, pp212-229, April, 1961. A paper describing simulated annealing isKirkpatrick, Gelatt, and Vecchi, “Optimization by Simulated Annealing,”Science 220 (1983) 671-680. Various other search techniques aredescribed in Reeves, ed., Modern Heuristic Techniques for CombinatorialProblems, Wiley (1993).

Step 2506 is implemented as a decision block. The search technique isoperated to identify a fixed number of candidate normals. If the searchprocess is not yet complete, the CPU 12 outputs a newly selected normalvalue, identified via the search process (step 2508). The CPU 12 thenproceeds to repeat steps 2502 and 2504, using the new candidate normal,to calculate an entropy measure for the image.

In the event that the search process is complete, the CPU 12 operates toselect and output the normal corresponding to the bin color distributionhaving the lowest entropy (showing the highest order for thechromaticity plane, and, thus, the most accurate projection of imagecolors).

Returning once again to FIG. 7 c, the output of the normal valuecorresponding to the lowest entropy completes step 2004. In step 2006,the CPU 12 outputs the optimized value for the one of the 1−K estimatednormals input to the routine of FIG. 7 c. As noted above, the CPU 12executes the routine of FIG. 7 c (step 1006 of FIG. 7 b) once for eachof the 1−K normals selected by a user and/or calculated by the CPU 12,to optimize each one of the respective 1−K normals relative to all ofthe pixels of the image being processed.

Upon completion of step 1006, the CPU 12 proceeds to step 1008. As shownin FIG. 7 b, step 1008 includes sub-steps 1008 a-c. In each sub-step1008 a-c, the CPU 12 operates to generate a normal image or map for oneor more of the optimized values for the 1−K normals. In each normal map,for each pixel (or Type C token (input in step 1004)) of the image I,the CPU 12 designates a normal value, for use in a projection to thechromaticity plane. The number of sub-steps can be 1−M, wherein M is avariable. For example, M can be set to equal K such that there is asingle normal map for each one of the optimized 1−K normals. In thealternative, there can be multiple normal maps, each based upon some orall of the 1−K normals.

In the simplest case, the CPU 12 assigns the same optimized one of the1−K normals for each pixel or Type C token in the respective map, withone map for each one of 1−K normals. The optimized one of the 1−Knormals can also be an average of all of the K normals to provide asingle normal map, with the average value for the normal assigned toeach pixel in the image. In other cases, additional maps can begenerated for the 1−K normals, including spatially varying normal valuesto accommodate varying conditions that can exist throughout an image(see the examples illustrated next to step 1008 c in FIG. 7 b).

For example, a normal map can contain a normal for each image location,each normal being based upon all or some of the 1−K normal estimates,such as the k nearest neighboring normals to an image location (pixel orType C token) (k-NN), where k is a number set at a value equal to all ora sub-set of the number of 1−K normals, and/or each normal beingcalculated using a linear system of equations based upon constraintsplaced upon relationships among the 1−K normals.

In a k-NN algorithm, the CPU 12 computes for each image location (pixelor Type C token), the distance between that location and each of the knearest normals to the respective location. The value for k is set to anumber equal to all or any sub-set of the 1−K number of optimized normalestimates found in step 1006 of FIG. 7 b. The CPU 12 then converts thecalculated distances into corresponding weights for use in computing anormal for each image location, as a function of the k nearest normalestimates.

In an exemplary embodiment of the present invention, the distancefunction for each image location (for example, each pixel p_(i)) to eachone of the k nearest normals to that location, is a weighted linearcombination of M individual distance functions, as follows:d(p _(i) ,P _(x))=Σ(from l=1 to M)d _(l)(p _(i) ,P _(x))wherein, similar to the optimization method of the routine of FIG. 7 c,M is set to 2, and d_(l) (p_(i), P_(x)) is the distance between aparticular pixel p, and one of the k nearest normals, P_(x) are thepixels used to estimate the one of the k nearest normals, and each d_(l)(p_(i), P_(x)) includes a spatial distance, d_(spatial)(p_(i), P_(x))and a spectral distance, d_(spectral)(p_(i), P_(x)), each calculated asin the optimization method.

Moreover, in the case of M=2, the weighted distance function for eachpixel p, is expressed, as follows:d(p _(i) ,P _(x))=λ₁ d _(spatial)(p _(i) ,P _(x))+λ₂ d _(spectral)(p_(i) ,P _(x))wherein λ₁ and λ₂ are parameters set to control the relative importancebetween the spatial and spectral distances in the weighted distancecalculation.

In the example of finding a normal based upon a k-NN algorithm when theimage locations are pixels p_(i), upon calculating a weighted distance d(p_(i), P_(x)) for each pixel p_(i), the CPU 12 can proceed to acomputation of a normal n_(i) for each pixel location, as follows:n _(i)=Σ(from x=1 to k)w _(x) n _(x)wherein n_(x) is one of the k nearest normals, and the weight for thepixel relative to the one normal, w_(x)=1−d (p_(i),P_(x))/max_(x=1 . . . k)d (p_(i), P_(x))+ε wherein d (p_(i), P_(x)) isthe weighted distance to the current one of the k nearest normals in thesummation, max_(x=1 . . . k)d(p_(i), P_(x)) is the distance to thefurthest one of the k normals and ε is a small weight added to thedistance to the furthest one of the normals to keep w_(x) non-zero, forexample, ε=0.01.

A normal map can then be output by the CPU 12 with an n_(i) normal valuedetermined as a function of the k-NN algorithm, for each pixel locationin the image I.

In a further exemplary embodiment of the present invention, the CPU 12executes one or more of sub-steps 1008 a-c of FIG. 7 b by building asystem of linear equations based upon a series of constraints between,for example, n, normal values for image locations determined as afunction of the k-NN algorithm, to generate a normal map. FIG. 7 f is aflow chart showing the use of a system of linear equations to estimatespatially varying normals for each token t in the image I.

In steps 600 a-d, information and parameters relevant to the linearequations are input to the CPU 12. For example, in step 600 a, a set ofvalues E represents an edge value e_(i), one value for each token in theimage, where e_(i)=1 if the corresponding token of the image overlaps amaterial edge, and e_(i)=0 if otherwise. In step 600 b a smoothnessweight w_(smooth)<<1 is input to the CPU 12.

In an exemplary embodiment of the present invention, the set of weightsE is determined based upon an edge detection. For example, normalsdetermined using the k-NN algorithm can be used to generate alog-chromaticity representation for the image, for example, by operatingthe CPU 12 to execute the routine of FIG. 8. The resultingrepresentation will be an illumination invariant version of the originalimage. A Canny edge detection algorithm can then be used to identifypixels forming an edge in the representation. Each edge pixel isassigned a 1 value. Then each token for the image I being processed(step 1004 of FIG. 7 b) including a majority of constituent pixels witha 1 value is assigned an e_(i)=1, and all other tokens are assigned ane_(i)=0.

Each of the set of values E and the weight w_(smooth) are used to definea set of weight values for use in smoothness constraints (step 602) suchthat a weight a, for a corresponding token is set to w_(smooth) when thee_(i) for the token is 1, and set to 1 when the e_(i) is 0. Thesmoothness constraint takes the form shown in step 604, as follows:

${\left\lbrack {\alpha_{i} - \alpha_{j}} \right\rbrack\begin{bmatrix}x_{i} \\x_{j}\end{bmatrix}} = 0$wherein x_(i) and x_(j) are normals to be determined for two adjacenttokens, the constraint encourages the two normals to be the same whenneither one of the adjacent tokens is an edge token.

In step 600 c, a data weight value, w_(data), is input to the CPU 12.The weight is used in a data constraint that encourages a normal for animage location to stay close to an estimate for the normal valuedetermined in a previously executed method, for example, the k-NNmethod. Then weight sets the relative importance of the data constraint.In step 606, a set of normal estimates V={v₁, v₂ . . . v_(p)}, is inputto the CPU 12, wherein each v_(i) is, for example, a k-NN estimate for anormal for an image token t. The constraint takes the form as shown instep 608: w_(data)x_(i)=v_(i), wherein x_(i) is the normal to bedetermined for the corresponding token t.

In step 600 d, a weight w_(anchor), for an anchor constraint, is inputto the CPU 12. The anchor constraint focuses on tokens that includepixels used to estimate the 1−K normals. To that end, in step 610, aninput to the CPU 12 includes the set of 1−K normals N, including all ofthe normals estimated by a user selection and/or calculated by the CPU12, and the associated token positions {T₁ . . . T_(N)}, each tokenposition including constituent pixels P corresponding to the pixels usedto calculate a respective one of the 1−K normals, as described above.The anchor constraint encourages tokens used to estimate a normal tostay at that normal value. The constraint takes the form as shown instep 612: w_(anchor)x_(i)=n_(i) wherein the weight w_(anchor) sets therelative importance of the anchor constraint, x_(i) is the normal to bedetermined for the corresponding token, and n_(i) is the one of the 1−Knormals estimates associated with the token.

In steps 614 and 616, the CPU 12 concatenates the left-hand andright-hand sides of the constraint equations of steps 604, 608 and 612,respectively, in an [L] [x]=[r] matrix equation. In step 614, the [L]matrix includes a concatenation of each left-hand side of each of thesmoothness, data and anchor constraints, with each instance of eachconstraint forming a row across the matrix. In step 616, the [r] matrixincludes a concatenation of each right hand side of each of thesmoothness, data and anchor constraints, with each instance of eachconstraint forming a row across the matrix.

In step 618, the an [A][x]=[b] matrix is built by the CPU 12 accordingto the following relationships: [A]=[L^(T)] [L] and [b]=[L^(T)] [r],wherein [L^(T)] is the transpose of [L] and [x] is a set of normalsN={x_(i) . . . x_(n)} to be determined by a solution to the matrixequation.

In step 620, the CPU 12 solves for [x], a matrix of optimized normals,one for each image location, for example, Type C tokens, wherein eachnormal is a vector, for example a 3-vector in the RGB color space of theexemplary embodiment of the present invention. The solution can beimplemented as a known least-squares algorithm.

In step 622, the CPU 12 outputs the set of normals N={x_(i) . . .x_(n)}.

FIG. 7 g shows an example of a token map having four tokens analyzed inthe system of linear equations of FIG. 7 f. In the example, four tokensa, b, c and d include one token (b) identified as an edge token, and onetoken (d) identified as a token used to estimate a normal n_(b). Thematrix shown in FIG. 7 g represents an application of smoothnessconstraints to the set of tokens a, b, c and d and an anchor constraintto the token d. The CPU 12 can solve the matrix equation to provide aset of normals, x_(a), x_(b), x_(c), and x_(d), one for each of thetokens a, b, c and d, respectively.

Upon completion of steps 1008 a-c of FIG. 7 b, the CPU 12 proceeds tosub-steps 1010 a-c. In each sub-step 1010 a-c, the CPU executes theroutines of FIGS. 8-11, to generate a log-chromaticity clustering mapcorresponding to a respective one of the M normal maps. Each clusteringmap provides a same reflectance map since the pixels or Type C tokens(step 1004) included in each identified cluster relate to a singlematerial reflectance, independent of illumination. Each cluster mapincludes a list of cluster group memberships cross-referenced to thepixels or Type C tokens of the image being processed I.

FIG. 8 is a flow chart for determining log-chromaticity coordinates forthe colors of an input image, as identified in steps 202 or 204 of theroutine of FIG. 6. In step 220, a list of colors is input to the CPU 12.The list of colors can comprise either the list generated throughexecution of step 202 of the routine of FIG. 6, or the list generatedthrough execution of step 204. In step 222, one of the maps for one ofthe optimized 1−K log-chromaticity orientations for the normal, N,determined through execution of the routine of FIG. 7 b, is also inputto the CPU 12.

In step 224, the CPU 12 operates to calculate a log value for each colorin the list of colors and plots the log values in a three dimensionallog space at respective (log R, log G, log B) coordinates, asillustrated in FIG. 5. Materials A, B and C denote log values forspecific colors from the list of colors input to the CPU 12 in step 220.A log-chromaticity plane is also calculated by the CPU 12, in the threedimensional log space, with u, v coordinates and an orientation set byN, input to the CPU 12 in step 222. The orientation N used for eachparticular color is set to correspond to the normal indicated for thecorresponding pixel or Type C token in the respective normal map. Eachu, v coordinate in the log-chromaticity plane can also be designated bya corresponding (log R, log G, log B) coordinate in the threedimensional log space.

According to a feature of the related invention, the CPU 12 thenprojects the log values for the colors A, B and C onto thelog-chromaticity plane (oriented, in each case, according to the normalN listed in the normal map for the pixel (or Type C token) correspondingto the color) to determine a u, v log-chromaticity coordinate for eachcolor. Each u, v log-chromaticity coordinate can be expressed by thecorresponding (log R, log G, log B) coordinate in the three dimensionallog space. The CPU 12 outputs a list of the log-chromaticity coordinatesin step 226. The list cross-references each color to a u, vlog-chromaticity coordinate and to the pixels (or a Type C token) havingthe respective color (depending upon the list of colors used in theanalysis (either step 202 (tokens) or 204 (pixels))).

FIG. 9 is a flow chart for optionally augmenting the log-chromaticitycoordinates for pixels or Type C tokens with extra dimensions, accordingto a feature of the related invention. In step 230, the list oflog-chromaticity coordinates, determined for the colors of the inputimage through execution of the routine of FIG. 8, is input to the CPU12. In step 232, the CPU 12 accesses the input image file 18, for use inthe augmentation.

In step 234, the CPU 12 optionally operates to augment eachlog-chromaticity coordinate with a tone mapping intensity for eachcorresponding pixel (or Type C token). The tone mapping intensity isdetermined using any known tone mapping technique. An augmentation withtone mapping intensity information provides a basis for clusteringpixels or tokens that are grouped according to both similarlog-chromaticity coordinates and similar tone mapping intensities. Thisimproves the accuracy of a clustering step.

In step 236, the CPU 12 optionally operates to augment eachlog-chromaticity coordinate with x, y coordinates for the correspondingpixel (or an average of the x, y coordinates for the constituent pixelsof a Type C token) (see FIG. 2 showing a P (1,1) to P (N, M) pixelarrangement). Thus, a clustering step with x, y coordinate informationwill provide groups in a spatially limited arrangement, when thatcharacteristic is desired.

In each of steps 234 and 236, the augmented information can, in eachcase, be weighted by a factor w₁ and w₂, w₃ respectively, to specify therelative importance and scale of the different dimensions in theaugmented coordinates. The weight factors w₁ and w₂, w₃ areuser-specified. Accordingly, the (log R, log G, log B) coordinates for apixel or Type C token is augmented to (log R, log G, log B, T*w₁, x*w₂,y*w₃) where T, x and y are the tone mapped intensity, the x coordinateand the y coordinate, respectively.

In step 238, the CPU 12 outputs a list of the augmented coordinates. Theaugmented log-chromaticity coordinates provide accurate illuminationinvariant representations of the pixels, or for a specified regionalarrangement of an input image, such as, for example, Type C tokens.According to a feature of the related invention and the presentinvention, the illumination invariant characteristic of thelog-chromaticity coordinates is relied upon as a basis to identifyregions of an image of a single material or reflectance, such as, forexample, Type B tokens.

FIG. 10 is a flow chart for clustering the log-chromaticity coordinates,according to a feature of the present invention. In step 240, the listof augmented log-chromaticity coordinates is input the CPU 12. In step242, the CPU 12 operates to cluster the log-chromaticity coordinates.According to the teachings of the related invention, the clustering stepcan be implemented via, for example, a known k-means clustering. Anyknown clustering technique can be used to cluster the log-chromaticitycoordinates to determine groups of similar log-chromaticity coordinatevalues, according to the related invention. According to the teachingsof each of the related invention and the present invention, the CPU 12correlates each log-chromaticity coordinate to the group to which therespective coordinate belongs.

According to a feature of the present invention, the clustering step 242is implemented as a function of an index of the type used in databasemanagement, for example, a hash index, a spatial hash index, b-trees orany other known index commonly used in a database management system. Byimplementing the clustering step 242 as a function of an index, thenumber of comparisons required to identify a cluster group for eachpixel or token of an image is minimized. Accordingly, the clusteringstep can be executed by the CPU 12 in a minimum amount of time, toexpedite the entire image process.

FIG. 10 a is an illustration of a grid for a spatial hash, according toa feature of an exemplary embodiment of the present invention. As shownin FIG. 10 a, a spatial hash divides an image being processed into agrid of buckets, each bucket being dimensioned to bespatialThresh×spatialThresh. The grid represents a histogram of the u, vlog-chromaticity values for the cluster groups. As each cluster iscreated, a reference to the cluster is placed in the appropriate bucketof the grid.

Each new pixel or token of the image being processed is placed in thegrid, in the bucket it would occupy, as if the item (pixel or token) wasa new group in the clustering process. The pixel or token is thenexamined relative to the clusters in, for example, a 3×3 grid of bucketssurrounding the bucket occupied by the item being examined. The item isadded to the cluster group within the 3×3 gird, for example, if the itemis within a threshold for a clusterMean.

The CPU 12 also operates to calculate a center for each group identifiedin the clustering step. For example, the CPU 12 can determine a centerfor each group relative to a (log R, log G, log B, log T) space.

In step 244, the CPU 12 outputs a list of the cluster group membershipsfor the log-chromaticity coordinates (cross referenced to either thecorresponding pixels or Type C tokens) and/or a list of cluster groupcenters.

Pursuant to a further feature of the present invention, the list ofcluster group memberships can be augmented with a user input of imagecharacteristics. For example, a user can specify pixels or regions ofthe image that are of the same material reflectance. The CPU 12 operatesto overlay the user specified pixels or regions of same reflectance ontothe clustering group membership information.

As noted above, in the execution of the clustering method, the CPU 12can use the list of colors from either the list generated throughexecution of step 202 of the routine of FIG. 6, or the list generatedthrough execution of step 204. In applying the identified cluster groupsto an input image, the CPU 12 can be operated to use the same set ofcolors as used in the clustering method (one of the list of colorscorresponding to step 202 or to the list of colors corresponding to step204), or apply a different set of colors (the other of the list ofcolors corresponding to step 202 or the list of colors corresponding tostep 204). If a different set of colors is used, the CPU 12 proceeds toexecute the routine of FIG. 11.

FIG. 11 is a flow chart for assigning the log-chromaticity coordinatesto clusters determined through execution of the routine of FIG. 10, whena different list of colors is used after the identification of thecluster groups, according to a feature of the present invention. In step250, the CPU 12 once again executes the routine of FIG. 8, this time inrespect to the new list of colors. For example, if the list of colorsgenerated in step 202 (colors based upon Type C tokens) was used toidentify the cluster groups, and the CPU 12 then operates to classifylog-chromaticity coordinates relative to cluster groups based upon thelist of colors generated in step 204 (colors based upon pixels), step250 of the routine of FIG. 11 is executed to determine thelog-chromaticity coordinates for the colors of the pixels in the inputimage file 18.

In step 252, the list of cluster centers is input to the CPU 12. In step254, the CPU 12 operates to classify each of the log-chromaticitycoordinates identified in step 250, according to the nearest clustergroup center. In step 256, the CPU 12 outputs a list of the clustergroup memberships for the log-chromaticity coordinates based upon thenew list of colors, with a cross reference to either correspondingpixels or Type C tokens, depending upon the list of colors used in step250 (the list of colors generated in step 202 or the list of colorsgenerated in step 204).

Upon completion of sub-steps 1010 a-c, the CPU 12 outputs a set oflog-chromaticity clustering maps, each including a list of cluster groupmemberships (from either steps 244 or 256 of the routines of FIGS. 8-11)for the pixels or Type C tokens of the image, and each one of thelog-chromaticity clustering maps corresponding to one of the normal mapsgenerated in sub-steps 1008 a-c. In step 1012, the CPU 12 operates tomerge the cluster group membership lists obtained from the multipleexecutions of the routines of FIGS. 8-11 in sub-steps 1010 a-c, into asingle composite cluster group membership list.

FIG. 7 h is a flow chart for a multi-clustering merge step to provide aroutine to implement step 1012 of FIG. 7 b. In step 700, the set oflog-chromaticity clustering maps generated in steps 1010 a-c of FIG. 7 bis input to the CPU 12. In a decision block (step 702) the CPU 12determines if the input set of log-chromaticity clustering maps includesmore than one map. In the simplest case, when K equals one optimizednormal, there can be a case when only one cluster map is generated. Inthat case, the CPU 12 returns the single log-chromaticity clustering mapas an output (step 704).

In the event the set of log-chromaticity clustering maps includes morethan one map, the CPU 12 proceeds to step 706. In step 706, the CPU 12operates to compute a score S for each cluster of each cluster map. Thescore is a measure of how likely a particular cluster includes lit andshadowed pixels of a single material reflectance. In an exemplaryembodiment of the present invention, the score is obtained in a two stepmethod applied to each cluster of each cluster map.

In a first step, the CPU 12 identifies the minimum and maximum RGBvalues, v₁ and v₂ among the pixels (or among the average color valuesfor Type C tokens) of a particular cluster. In the second step, the CPU12 computes the Euclidean distance between v₁ and v₂ as the score forthe respective cluster.

In step 708, the CPU 12 generates a single cluster map by examining thecluster membership for each <x, y> location in the image plane (see, forexample, FIG. 2) (i.e. the cluster group wherein the u, v coordinates ofthe chromaticity plane corresponding to the <x, y> location, has beenplaced), as indicated in each one of the set of log-chromaticityclustering maps, to select the one membership cluster for the respective<x, y> location, from among the entire set of log-chromaticityclustering maps, having the highest score. The single resulting clustermap therefor includes, as a cluster membership for each <x, y> location,the cluster for that location with the highest score, indicating thecluster for the location having the highest likelihood of including litand shadowed pixels of a single material reflectance, and thus,providing the most accurate log-chromaticity representation for eachrespective pixel.

Referring once again to FIG. 7 b, in step 1014, the CPU 12 outputs thesingle cluster map.

FIG. 12 is a flow chart for detecting regions of uniform reflectancebased on the log-chromaticity clustering according to a feature of thepresent invention. In step 260, the input image file 18 is once againprovided to the CPU 12. In step 262, one of the pixels or Type C tokens,depending upon the list of colors used in step 250, is input to the CPU12. In step 264, the cluster membership information contained in thesingle composite cluster map, obtained from the merge performed in FIG.7 h, is input to the CPU 12.

In step 266, the CPU 12 operates to merge each of the pixels, orspecified regions of an input image, such as, for example, Type Ctokens, having a same cluster group membership into a single region ofthe image to represent a region of uniform reflectance (Type B token).The CPU 12 performs such a merge operation for all of the pixels ortokens, as the case may be, for the input image file 18. In step 268,the CPU 12 outputs a list of all regions of uniform reflectance (andalso of similar tone mapping intensities and x, y coordinates, if thelog-chromaticity coordinates were augmented in steps 234 and/or 236). Itshould be noted that each region of uniform reflectance (Type B token)determined according to the features of the present invention,potentially has significant illumination variation across the region.

U.S. Pat. No. 8,139,867 teaches a constraint/solver model forsegregating illumination and material in an image, including anoptimized solution based upon a same material constraint. A samematerial constraint, as taught in U.S. Pat. No. 8,139,867, utilizes TypeC tokens and Type B tokens, as can be determined according to theteachings of the present invention. The constraining relationship isthat all Type C tokens that are part of the same Type B token areconstrained to be of the same material. This constraint enforces thedefinition of a Type B token, that is, a connected image regioncomprising contiguous pixels that represent a region of the imageencompassing a single material in the scene, though not necessarily themaximal region corresponding to that material. Thus, all Type C tokensthat lie within the same Type B token are by the definition imposed uponType B tokens, of the same material, though not necessarily of the sameillumination. The Type C tokens are therefore constrained to correspondto observed differences in appearance that are caused by varyingillumination.

FIG. 13 is a representation of an [A] [x]=[b] matrix relationship usedto identify and separate illumination and material aspects of an image,according to a same-material constraint, as taught in U.S. Pat. No.8,139,867. Based upon the basic equation I=ML (I=the recorded imagevalue, as stored in an image file 18, M=material reflectance, andL=illumination), log(I)=log(ML)=log(M)+log(L). This can be restated asi=m+l, wherein i represents log(I), m represents log(M) and l representslog(L). In the constraining relationship of a same material, in anexample where three Type C tokens, a, b and c, (as shown in FIG. 13) arewithin a region of single reflectance, as defined by a correspondingType B token defined by a, b and c, then m_(a)=m_(b)=m_(c). For thepurpose of this example, the I value for each Type C token is theaverage color value for the recorded color values of the constituentpixels of the token. The a, b and c, Type C tokens of the example cancorrespond to the blue Type B token illustrated in FIG. 3 d.

Since: m_(a)=i_(a)−l_(a), m_(b)=i_(b)−l_(b), and m_(c)=i_(c)−l_(c),these mathematical relationships can be expressed, in a same materialconstraint, as (1)l_(a)+(−1)l_(b)+(0)l_(c)=(i_(a)−i_(b)),(1)l_(a)+(0)l_(b)+(−1)l_(c)=(i_(a)−i_(c)) and(0)l_(a)+(1)l_(b)+(−1)l_(c)=(i_(b)−i_(c)).

Thus, in the matrix equation of FIG. 13, the various values for thelog(I) (i_(a), i_(b), i_(c)), in the [b] matrix, are known from theaverage recorded pixel color values for the constituent pixels of theadjacent Type C tokens a, b and c. The [A] matrix of 0's, 1's and −1's,is defined by the set of equations expressing the same materialconstraint, as described above. The number of rows in the [A] matrix,from top to bottom, corresponds to the number of actual constraintsimposed on the tokens, in this case three, the same material constraintbetween the three adjacent Type C tokens a, b and c. The number ofcolumns in the [A] matrix, from left to right, corresponds to the numberof unknowns to be solved for, again, in this case, the threeillumination values for the three tokens. Therefore, the values for theillumination components of each Type C token a, b and c, in the [x]matrix, can be solved for in the matrix equation, by the CPU 12. Itshould be noted that each value is either a vector of three valuescorresponding to the color bands (such as red, green, and blue) of ourexample or can be a single value, such as in a grayscale image.

Once the illumination values are known, the material color can becalculated by the CPU 12 using the I=ML equation. Intrinsic illuminationand material images can be now be generated for the region defined bytokens a, b and c, by replacing each pixel in the original image by thecalculated illumination values and material values, respectively. Anexample of an illumination image and material image, corresponding tothe original image shown in FIG. 3 b, is illustrated in FIG. 14.

Implementation of the constraint/solver model according to thetechniques and teachings of U.S. Pat. No. 8,139,867, utilizing the TypeC tokens and Type B tokens obtained via a log-chromaticity clusteringtechnique according to the present invention, provides a highlyeffective and efficient method for generating intrinsic imagescorresponding to an original input image. The intrinsic images can beused to enhance the accuracy and efficiency of image processing, imageanalysis and computer vision applications.

However, the intrinsic images generated from the performance of theexemplary embodiments of the present invention can include artifactsthat distort the appearance of a scene depicted in the image beingprocessed. The artifacts can be introduced through execution of theintrinsic image generations methods of the present invention, or throughuser modifications such as the user input of image characteristicsdiscussed above. Accordingly, according to a feature of the presentinvention, various post processing techniques can be implemented toreduce the artifacts.

FIG. 15 is a flow chart for an edge preserving blur post processingtechnique applied to the intrinsic images illustrated in FIG. 14,according to a feature of the present invention, to improve the qualityof the illumination and material reflectance aspects depicted in theintrinsic images. In step 300, the CPU 12 receives as an input anoriginal image (an image file 18), and the corresponding intrinsicmaterial reflectance and illumination images determined by the CPU 12through solution of the matrix equation shown in FIG. 13, as describedabove.

In step 302, the CPU 12 operates to perform an edge-preserving blur ofthe illumination in the illumination image by applying an edgepreserving smoothing filter. The edge preserving smoothing filter can beany one of the known filters such as, for example, a bilateral filter, aguided filter, a mean-shift filter, a median filter, anisotropicdiffusion and so on. The filter can be applied one or more times to theillumination image. In an exemplary embodiment, a bilateral filter isapplied to the illumination image twice. In addition, several differenttypes of filters can be applied in succession, for example, a medianfilter followed by a bilateral filter.

In step 304, the CPU 12 recalculates the intrinsic material reflectanceimage based upon the I=ML equation, and using the original image of theimage file 18 and the illumination image, as modified in step 302. Instep 306, the CPU 12 outputs intrinsic material reflectance andillumination images, as modified by the CPU 12 through execution of theroutine of FIG. 15.

A smoothing filter applied to the illumination image results in severalimprovements to the appearance of the intrinsic images when used in, forexample, such applications as computer graphics. For example, incomputer graphics, texture mapping is used to achieve certain specialeffects. Artists consider it desirable in the performance of texturemapping to have some fine scale texture form the illumination in thematerial reflectance image. By smoothing the illumination image, in step302, the fine scale texture is moved to the material reflectance imageupon a recalculation of the material image in step 304, as will bedescribed below.

In addition, smoothing the illumination in step 302 places some of theshading illumination (illumination intensity variation due to curvatureof a surface) back into the material reflectance image, giving thematerial image some expression of curvature. That results in an improvedmaterial depiction more suitable for artistic rendering in a computergraphics application.

Moreover, small reflectance variation sometimes erroneously ends up inthe illumination image. The smoothing in step 302 forces the reflectancevariation back into the material image.

FIG. 16 is a flow chart for an artifact reduction post processingtechnique applied to the intrinsic images illustrated in FIG. 14,according to a feature of the present invention, to improve the qualityof the illumination and material reflectance aspects depicted in theintrinsic images. In step 400, the CPU 12 receives as an input anoriginal image (an image file 18), and the corresponding intrinsicmaterial reflectance and illumination images determined by the CPU 12through solution of the matrix equation shown in FIG. 13, as describedabove. Optionally, the intrinsic images can be previously modified bythe CPU 12 through execution of the routine of FIG. 15.

In step 402, the CPU 12 operates to calculate derivatives (thedifferences between adjacent pixels) for the pixels of each of theoriginal image and the material reflectance image. Variations betweenadjacent pixels, in the horizontal and vertical directions, are causedby varying illumination and different materials in the scene depicted inthe original image. When the CPU 12 operates to factor the originalimage into intrinsic illumination and material reflectance images, someof the variation ends up in the illumination image and some ends up inthe material reflectance image. Ideally, all of the variation in theillumination image is attributable to varying illumination, and all ofthe variation in the material reflectance image is attributable todifferent materials.

Thus, by removing the illumination variation, variations in the materialreflectance image should be strictly less than variations in theoriginal image. However, inaccuracies in the process for generating theintrinsic images can result in new edges appearing in the materialreflectance image.

In step 404, the CPU 12 operates to identify the artifacts caused by thenewly appearing edges by comparing the derivatives for the materialreflectance image with the derivatives for the original image. The CPU12 modifies the derivatives in the material reflectance image such that,for each derivative of the material reflectance image, the sign ispreserved, but the magnitude is set at the minimum of the magnitude ofthe derivative in the original image and the material reflectance image.The modification can be expressed by the following equation:derivativeReflectanceNew=min(abs(derivativeReflectanceOld),abs(derivativeOriginalimage))*sign(derivativeReflectanceOld)

In step 406, the CPU integrates the modified derivatives to calculate anew material reflectance image. The new image is a material reflectanceimage without the newly appearing, artifact-causing edges. Any knowntechnique can be implemented to perform the integration. For example,the CPU 12 can operate to perform numerical 2D integration by solvingthe 2D Poisson equation using discrete cosine transforms.

In step 408, the CPU 12 recalculates the intrinsic illumination imagebased upon the I=ML equation, and using the original image of the imagefile 18 and the material reflectance image, as modified in steps 404 and406. In step 408, the CPU 12 outputs intrinsic material reflectance andillumination images, as modified by the CPU 12 through execution of theroutine of FIG. 16.

FIG. 17 is a flow chart for a BIDR model enforcement post processingtechnique applied to the intrinsic images illustrated in FIG. 14,according to a feature of the present invention, to improve the qualityof the illumination and material reflectance aspects depicted in theintrinsic images.

As described above, the BIDR model predicts the correct color for amaterial, in a shadow penumbra, from full shadow to fully lit. As shownin FIG. 18, according to the prediction of the BIDR model, colors for amaterial, for example, in an RGB color space, from a fully shaded colorvalue to a fully lit color value, generally form a line in the colorspace. In full shadow, the material is illuminated by an ambientilluminant, while when fully lit, the material is illuminated by theambient illuminant and the direct or incident illuminant present in thescene at the time the digital image of an image file 18 was recorded.

According to the BIDR model, the illumination values in an image alsodefine a line extending from the color of the ambient illuminant to thecolor of the combined ambient and direct illuminants. In log colorspace, the illumination line predicted by the BIDR model corresponds tothe normal, N of the log color space chromaticity plane illustrated inFIG. 5.

Various inaccuracies in the generation of the illumination and materialintrinsic images, as described above, can also result, for example, inillumination values in the generated intrinsic illumination image thatdiverge from the line for the illumination values predicted by the BIDRmodel. According to the present invention, the illumination lineprediction of the BIDR model is used to correct such inaccuracies bymodifying the illumination to be linear in log(RGB) space.

Referring once again to FIG. 17, in step 500, the CPU 12 receives asinput a BIDR illumination orientation, corresponding to the normal Nillustrated in FIG. 5. In the exemplary embodiment of the presentinvention, N is determined by the CPU 12 through execution of theroutine of FIG. 7, as described above. In that case, the N determinedthrough execution of the routine of FIG. 7 is used in both theclustering process described above, and in the BIDR model enforcementpost processing technique illustrated in FIG. 17.

In the event the illumination and material reflectance images aregenerated via a method different from the log-chromaticity clusteringtechnique of the exemplary embodiment, the orientation N is determinedby the CPU 12 in a separate step before the execution of the routine ofFIG. 17, through execution of the routine of FIG. 7. When N isdetermined in a separate step, the CPU 12 can operate relative to eitherthe original image or the illumination image. In addition, when theprocessing is based upon a user input, as described above, the user canmake a selection from either the original image or the illuminationimage.

Moreover, in step 500, the CPU 12 also receives as input an originalimage (an image file 18), and the corresponding intrinsic materialreflectance and illumination images determined by the CPU 12 throughsolution of the matrix equation shown in FIG. 13, also as describedabove. Optionally, the intrinsic images can be previously modified bythe CPU 12 through execution of the routine(s) of either one, or bothFIGS. 15 and 16.

In step 502, the CPU 12 determines the full illumination color in theillumination image. The full illumination color (ambient+direct) can bethe brightest color value depicted in the illumination image. However,the brightest value can be inaccurate due to noise in the image or otheroutliers. In a preferred exemplary embodiment of the present invention,a more accurate determination is made by finding all illumination colorvalues in a preselected range of percentiles of the intensities, forexample, the 87^(th) through 92^(nd) percentiles, and calculating anaverage of those values. The average is used as the full illuminationcolor value. Such an approach provides a robust estimate of the brightend of the illumination variation in the intrinsic illumination image.

In step 504, the CPU 12 operates to modify all of the pixels of theillumination image by projecting all of the illumination colors depictedby the pixels in the illumination image to the nearest point on a linehaving the orientation N (input to the CPU 12 in step 500) and passingthrough the full illumination color determined in step 302. Thus, thecolor of each pixel of the illumination image is modified to conform tothe closest value required by the BIDR model prediction.

A special case exists for the pixels of the illumination image having anintensity that is greater than the full illumination color value, ascalculated in step 502. The special case can be handled by the CPU 12according to a number of different methods. In a first method, themodification is completed as with all the other pixels, by projectingeach high intensity pixel to the nearest value on the illumination line.In a second method, each high intensity pixel is replaced by a pixel setat the full illumination color value. According to a third method, eachhigh intensity pixel is kept at the color value as in the originalimage.

An additional method is implemented by using a weighted average for eachhigh intensity pixel, of values determined according to the first andthird methods, or of values determined according to the second and thirdmethods. The weights would favor values calculated according to eitherthe first or second methods when the values are similar to highintensity pixels that are not significantly brighter than the fullillumination color value calculated in step 502. Values calculated viathe third method are favored when values for high intensity pixels thatare significantly brighter than the full illumination color value. Sucha weighting scheme is useful when the I=ML equation for imagecharacteristics is inaccurate, for example, in the presence of specularreflections.

Any known technique can be implemented to determine the relative weightsfor illumination values. In an exemplary embodiment of the presentinvention, a sigmoid function is constructed such that all the weight ison the value determined either according to the first or second methods,when the intensity of the high intensity pixel is at or near the fullillumination color value, with a smooth transition to an equallyweighted value, between a value determined either according to the firstor second methods and a value determined according to the third method,as the intensity increases. That is followed by a further smoothtransition to full weight on a value determined according to the thirdmethod, as the intensity increase significantly beyond the fullillumination color value.

In step 506, the CPU 12 recalculates the intrinsic material reflectanceimage based upon the I=ML equation, and using the original image of theimage file 18 and the illumination image, as modified in step 504. Instep 508, the CPU 12 outputs intrinsic material reflectance andillumination images modified to strictly adhere to the predictions ofthe BIDR model.

For best results, the above-described post processing techniques can beexecuted in a log(RGB) space. Also, the various techniques can beexecuted in the order described above. Once one or more of the postprocessing techniques have been executed, the final modified intrinsicimages can be white balanced and/or scaled, as desired, and output bythe CPU 12.

In the preceding specification, the invention has been described withreference to specific exemplary embodiments and examples thereof. Itwill, however, be evident that various modifications and changes may bemade thereto without departing from the broader spirit and scope of theinvention as set forth in the claims that follow. The specification anddrawings are accordingly to be regarded in an illustrative manner ratherthan a restrictive sense.

What is claimed is:
 1. An automated, computerized method for processingan image, comprising the steps of: providing an image file depicting animage defined by image locations, in a computer memory; generating abi-illuminant chromaticity plane in a log color space for representingthe image locations of the image in a log-chromaticity representationfor the image; providing a set of estimates for an orientation of thebi-illuminant chromaticity plane; generating a plurality of normal maps,each normal map providing an indication of at least one orientation,each at least one orientation calculated as a function of the set ofestimates for an orientation; and calculating a set of log-chromaticitycluster maps, each one of the set of log-chromaticity cluster maps beingbased upon a corresponding one of the plurality of normal maps.
 2. Themethod of claim 1 wherein the image locations comprise pixels.
 3. Themethod of claim 1 wherein the image locations comprise tokens.
 4. Anautomated, computerized method for processing an image, comprising thesteps of: providing an image file depicting an image defined by imagelocations, in a computer memory; generating a bi-illuminant chromaticityplane in a log color space for representing the image locations of theimage in a log-chromaticity representation for the image; providing aset of estimates for an orientation of the bi-illuminant chromaticityplane; and generating a plurality of normal maps, each normal mapproviding an indication of at least one orientation, each at least oneorientation calculated as a function of the set of estimates for anorientation, wherein the step of generating a plurality of normal mapsis carried out by executing a technique based upon at least one of ak-NN algorithm and a system of linear equations representing constraintsbetween values based upon selected ones of the set of estimates for anorientation, relative to selected ones of the plurality of normal maps.5. A computer system which comprises: a CPU; and a memory storing animage file containing an image defined by image locations; the CPUarranged and configured to execute a routine to: generate abi-illuminant chromaticity plane in a log color space for representingthe image locations of the image in a log-chromaticity representationfor the image, provide a set of estimates for an orientation of thebi-illuminant chromaticity plan, generate a plurality of normal maps,each normal map providing an indication of at least one orientation,each at least one orientation calculated as a function of the set ofestimates for an orientation, and calculating a set of log-chromaticitycluster maps, each one of the set of log-chromaticity cluster maps beingbased upon a corresponding one of the plurality of normal maps.
 6. Acomputer program product, disposed on a non-transitory computer readablemedia, the product including computer executable process steps operableto control a computer to: provide an image file depicting an imagedefined by image locations, in a computer memory, generate abi-illuminant chromaticity plane in a log color space for representingthe image locations of the image in a log-chromaticity representationfor the image, provide a set of estimates for an orientation of thebi-illuminant chromaticity plane, generate a plurality of normal maps,each normal map providing an indication of at least one orientation,each at least one orientation calculated as a function of the set ofestimates for an orientation, and calculate a set of log-chromaticitycluster maps, each one of the set of log-chromaticity cluster maps beingbased upon a corresponding one of the plurality of normal maps.
 7. Thecomputer program product of claim 6 wherein the image locations comprisepixels.
 8. The computer program product of claim 6 wherein the imagelocations comprise tokens.
 9. A computer program product, disposed on anon-transitory computer readable media, the product including computerexecutable process steps operable to control a computer to: provide animage file depicting an image defined by image locations, in a computermemory, generate a bi-illuminant chromaticity plane in a log color spacefor representing the image locations of the image in a log-chromaticityrepresentation for the image, provide a set of estimates for anorientation of the bi-illuminant chromaticity plane, and generate aplurality of normal maps, each normal map providing an indication of atleast one orientation, each at least one orientation calculated as afunction of the set of estimates for an orientation, wherein the processstep to generate a plurality of normal maps is carried out by executinga technique based upon at least one of a k-NN algorithm and a system oflinear equations representing constraints between values based uponselected ones of the set of estimates for an orientation, relative toselected ones of the plurality of normal maps.
 10. A computer programproduct, disposed on a non-transitory computer readable media, theproduct including computer executable process steps operable to controla computer to: provide an image file depicting an image defined by imagelocations, in a computer memory, generate a bi-illuminant chromaticityplane in a log color space for representing the image locations of theimage in a log-chromaticity representation for the image, provide a setof estimates for an orientation of the bi-illuminant chromaticity plane,the set of estimates provided by user input, calculation based upon atleast one of a linear token, an x-junction and a Type B token, orcombinations thereof, and generate a plurality of normal maps, eachnormal map providing an indication of at least one orientation, each atleast one orientation calculated as a function of the set of estimatesfor an orientation; and calculate a set of log-chromaticity clustermaps, each one of the set of log-chromaticity cluster maps being basedupon a corresponding one of the plurality of normal maps.
 11. A computerprogram product, disposed on a non-transitory computer readable media,the product including computer executable process steps operable tocontrol a computer to: provide an image file depicting an image definedby image locations, in a computer memory, generate a bi-illuminantchromaticity plane in a log color space for representing the imagelocations of the image in a log-chromaticity representation for theimage, provide a set of estimates for an orientation of thebi-illuminant chromaticity plane, each one of the set of estimates beingbased upon one of a set of subsets of the image locations, execute adistance function between each one of the image locations and at leastone selected one of the sub-sets of the image locations used tocalculate the set of estimates to obtain a distance function result, andfor each one of the image locations, calculate an orientation based uponthe set of estimates and the distance function result.
 12. The computerprogram product of claim 11 wherein the distance function is based upona spatial distance and a spectral distance between each one of the imagelocations and the at least one selected one of the sub-sets of the imagelocations.